Bibliographical Notes on Tree-structured classifiers

The main source for the lecture on trees is Chapter 8, sections 1 to 4 inclusive, of the textbook [5]. CART is described in the lovely book by Breiman, Friedman, Olshen and Stone [1], which contains an extensive discussion of all the main points we touched during the lecture. A machine-learning point of view is contained in Chapter 3 of Mitchell’s book [8], which is a good reference for ID3 and C4.5. Devroye, Györfi, and Lugosi devote Chapters 20 and 21 of their book [4] to the topic. As usual, they devote a substantial effort to consistency proofs for classifiers, and provide us with interesting examples. Unlike the previous references, they also consider trees constructed using only the observations and not their labels. The material described in class came from two papers [3, 6]. The problem of matching impurity functions to loss functions was addressed by P.A. Chou in 1991 [3]. From this work we derived the analysis of the selection of optimal labels for nodes and leaves, and the theory of how to optimally choose splitting points. Chou’s paper also descries an algorithm that performs optimal (greedy) splitting in linear time (in the number of samples, in the number of classes and in the number of dimensions). The description of pruning methods provided during the lecture comes essentially from a paper by Esposito, et al.,[6]. Other pruning methods are also described in this work and in [7]. Two interesting interesting conclusions were reached in [6]: the use of a pruning set is often a bad idea, and some common methods exhibit a marked tendency to overprune/underprune, which one would not easily infer from their description.